Huawei has always been committed to exploring the AI application in historical research. Biography generation, as a specialized form of abstractive summarization, plays a crucial role in historical research but faces unique challenges that existing large language models (LLMs) struggle to address. These challenges include maintaining stylistic adherence to historical writing conventions, ensuring factual fidelity, and handling fragmented information across multiple documents. We present AIstorian, a novel end-to-end agentic system featured with a knowledge graph (KG)-powered retrieval-augmented generation (RAG) and anti-hallucination multi-agents. Specifically, AIstorian introduces an in-context learning based chunking strategy and a KG-based index for accurate and efficient reference retrieval. Meanwhile, AIstorian orchestrates multi-agents to conduct on-the-fly hallucination detection and error-type-aware correction. Additionally, to teach LLMs a certain language style, we finetune LLMs based on a two-step training approach combining data augmentation-enhanced supervised fine-tuning with stylistic preference optimization. Extensive experiments on a real-life historical Jinshi dataset demonstrate that AIstorian achieves a 3.8x improvement in factual accuracy and a 47.6% reduction in hallucination rate compared to existing baselines. The data and code are available at: https://github.com/ZJU-DAILY/AIstorian.

Recent studies have identified one aggravating factor of LLM hallucinations as the knowledge inconsistency between pre-training and fine-tuning, where unfamiliar fine-tuning data mislead the LLM to fabricate plausible but wrong outputs. It also opens new possibilities for knowledge-controlled generation in LLMs. Hallucination of large language models (LLMs) refers to the phenomenon where LLMs' outputs look plausible but diverge from real-world facts. It has become a major concern of LLMs, seriously undermining their reliability and trustworthiness (Huang et al., 2023; Ji et al., 2023). Recent research has unveiled one aggravating factor of LLM hallucination, which is the knowledge inconsistency between the pre-training and tuning (e.g., instruction-or fine-tuning) stages (Gekhman et al., 2024; Kang et al., 2024; Lin et al., 2024). More specifically, if the tuning stage involves training examples that require knowledge that an LLM has not seen during pre-training, then the LLM would be misled to fabricate plausible but wrong answers to unfamiliar questions (Schulman, 2023; Gao, 2021; Goldberg, 2023). For example, consider fine-tuning a model for a question answering (QA) task with the example'When was John Estes born?' and assume that the LLM has never learned about John Estes during pre-training. However, since the LLM is still trained to produce the correct answer, '1987', it is consequently encouraged to respond with a random legitimate year whenever it is asked about the birth year of any unknown person, thus giving rise to hallucination. These findings highlight an important but previously understudied consideration of LLM training, which is the disentanglement between knowledge and skill. Specifically, it is discovered that knowledge and skills are acquired at different stages of LLM training, the former at pre-training, and the latter at tuning (Zhou et al., 2023; Gudibande et al., 2024). However, although the focus in the tuning stage is to learn skills, not knowledge, the learning process is still interfered with by any inconsistency in the knowledge aspect, because the information on the two aspects is entangled.